<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
	  "http://www.w3.org/TR/html4/loose.dtd">
<html>
  <head>
    <title>The deal.II Readme on interfacing to PETSc</title>
    <link href="../screen.css" rel="StyleSheet">
    <meta name="copyright" content="Copyright (C) 2008 - 2020 by the deal.II authors">
    <meta name="keywords" content="deal.II">
  </head>

  <body>

    <h1>Interfacing <acronym>deal.II</acronym> to PETSc</h1>

    <p>
      <a href="http://www.mcs.anl.gov/petsc/"
      target="_top">PETSc</a> is a
      software package that provides lots of functionality for linear
      algebra, among other things. For example, it includes implementations of a variety of
      linear solvers, as well as various different sparse and dense matrix and
      vector formats. Of particular interest to deal.II is their ability to
      provide this functionality both on sequential and parallel (using MPI)
      computers.
    </p>

    <p>
      <acronym>deal.II</acronym> has wrapper classes to the linear algebra
      parts of PETSc that provide almost the
      same interfaces as the built-in <acronym>deal.II</acronym> linear
      algebra classes. We use these interfaces for parallel computations based
      on MPI since the native deal.II linear algebra classes lack this
      ability. They are used, among other programs, in step-17, step-18 and
      step-40.
    </p>

    <h4>Installing <acronym>deal.II</acronym> with PETSc</h4>

    <p style="color: red"><b>Note:</b> The most recent version of PETSc
      that has been reported to be compatible with
      <acronym>deal.II</acronym> is version 3.13.1. If you use a later
      version than this and encounter problems, let us
      know. <acronym>deal.II</acronym> does not support versions of PETSc prior
      to 3.3.0.
    </p>

    <p>
      When you compile and install PETSc, you need to set
      environment variables <code>PETSC_DIR</code> and <code>PETSC_ARCH</code>
      to a path to PETSc and denoting the architecture for which PETSc is
      compiled. <code>PETSC_ARCH</code> is in reality just a name you give to
      your installation, it is a string you can choose however you like. The
      point of it is that it allows you to have multiple possibly different
      PETSc installations. A consequence of this is that you need to
      let <acronym>deal.II</acronym>'s <code>cmake</code> scripts know which
      one of these installations you want it to use, i.e., you need to set the
      <code>PETSC_ARCH</code> variable to the same value you used when you
      installed PETSc. The same is true for <code>PETSC_DIR</code>. You can
      this via environment variables. <code>cmake</code> will then also
      recognize that PETSc shall be used, and enable the wrapper classes,
      without you having to explicitly say that you want to use PETSc.
    </p>

    <p>
      Alternatively, the <code>-DPETSC_DIR=DIR</code> and
      <code>-DPETSC_ARCH=ARCH</code> options for <code>cmake</code>
      can be used to override the values of <code>PETSC_DIR</code>
      and <code>PETSC_ARCH</code> or if these environment
      variables are not set at all. If you do have a PETSc
      installation and have set the <code>PETSC_DIR</code> and
      <code>PETSC_ARCH</code> environment variables but do not wish
      <acronym>deal.II</acronym> to be configured for PETSc use, you
      should specify <code>-DDEAL_II_WITH_PETSC=OFF</code> as a flag
      during configuration.
    </p>

    <p><b>Note:</b> <acronym>deal.II</acronym> can be installed with both
      PETSc and Trilinos and they do not usually get in their
      respective ways. There are, however, occasions where this is not true
      and this fundamentally comes from the fact that both of these packages
      are built from subpackages that are developed by independent
      groups. Unfortunately, some of these sub-packages can be configured to
      be part of both PETSc and Trilinos, and if you try to
      use <acronym>deal.II</acronym> with versions of PETSc and Trilinos
      that <i>both</i> contain a particular sub-package, little good will come
      of it in general. In particular, we have experienced this with the ML
      package that can serve as an algebraic multigrid method to both PETSc
      and Trilinos. If both of these packages are configured to use ML, then
      difficult to understand error messages at compile or link time are
      almost inevitable, and there is little the <acronym>deal.II</acronym>
      build system can do to prevent this. Thus, <i>don't try to do that!</i>
    </p>


    <h4>Installing PETSc</h4>


    <p>
      Installing PETSc correctly can be a bit of a
      challenge. To start, take a look at
      the <a href="http://www.mcs.anl.gov/petsc/documentation/installation.html"
      target="_top">PETSc installation instructions</a>. We have found that
      the following steps generally appear to work where we simply unpack and
      build PETSc in its final location (i.e., we do not first build and then
      install it into a separate directory):
      <pre>

	tar xvzf petsc-x-y-z.tar.gz
        cd petsc-x-y-z
	export PETSC_DIR=`pwd`
	export PETSC_ARCH=x86_64       # or any other identifying text for your machine
	./config/configure.py --with-shared=1 --with-x=0 --with-mpi=1 --download-hypre=1
	make
      </pre>
    </p>

    <p>
      This automatically builds PETSc with both MPI and the algebraic
      multigrid preconditioner package Hypre (which we use in step-40). If you
      would like to use PETSc with MPI then we recommend that you install MPI
      through your package manager instead of letting PETSc install it: put
      another way, installing PETSc with the flag <code>--download-mpich</code>
      often causes problems (such as linking errors or poor performance)
      that may be avoided by using whatever your system provides instead.
      <br>
      Now let PETSc check his own sanity:
      <pre>

	make test
      </pre>
      will self-check the serial (and MPI) implementation of PETSc.
      <br>
      You may wish to put the <code>export</code> commands into
      your <code>~/.bashrc</code> or <code>~/.cshrc</code> files, with
      the first one replaced by something of the kind
      <pre>

	export PETSC_DIR=/path/to/petsc-x-y-z
      </pre>
    </p>

    <p>
      By default, PETSc is compiled in "debug mode". You can switch this to
      "optimized mode" by adding the command line parameter
      <pre>

	--with-debugging=0
      </pre>
      to the call of <code>./config/configure.py</code> above. In some cases,
      this has made linear solvers run up to 30% faster. As with choosing
      between <acronym>deal.II</acronym>'s debug and optimized modes, you
      should only use optimized PETSc builds once you have tested that your
      program runs well in debug mode.
    </p>

    <hr />
    <div class="right">
      <a href="http://validator.w3.org/check?uri=referer" target="_top">
        <img style="border:0" src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!"></a>
      <a href="http://jigsaw.w3.org/css-validator/check/referer" target="_top">
        <img style="border:0;width:88px;height:31px" src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
    </div>
  </body>
</html>
